Rescaling citations of publications in Physics 
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We analyze the citation distributions of all papers published in Physical Review journals between 
1985 and 2009. The average number of citations received by papers published in a given year and in 
a given field is computed. Large variations are found, showing that it is not fair to compare citation 
numbers across fields and years. However, when a rescaling procedure by the average is used, it 
is possible to compare impartially articles across years and fields. We make the rescaling factors 
available, for use by the readers. We also show that rescaling citation numbers by the number of 
publication authors has strong effects and should therefore be taken into account when assessing 
the bibliometric performance of researchers. 



PACS numbers: 01.30.-y, 01.40.G-, 01.78.+p 

I. INTRODUCTION 

Despite its many shortcomings, citation analy- 
sis is increasingly used as a quantitative tool to 
evaluate research performance, ranging from the 
single publication up to the level of individual re- 
searchers, groups, departments, institutions and 
countries [1-8 . The underlying idea is that the 
number of citations measures the impact of a pub- 
lication and it is thus a proxy for the quality and 
importance of the scientific work described in it. 
This assumption is highly questionable and the 
empirical evidence revealing striking counterexam- 
ples to its general validity is substantial [9HTT] . In 
principle, a careful, unbiased review by peers with 
no conflict of interest would be a better procedure 
to evaluate research quality and impact (although 
many distortions can occur also in this case [12]). 
Nevertheless, for several reasons that go much be- 
yond the scope of this work and will not be dis- 
cussed here, the use of quantitative bibliometric 
measures is not going to diminish in the future. It 
is much more realistic and effective trying to cor- 
rect the inadequacies of how citation analysis is 
performed, rather than self-deceptively hope that 
citation analysis will lose importance in the fu- 
ture [13]. 

In this paper, we tackle two of the more obvi- 
ous problems in the use of the number of citations 
to measure the impact of scientific publications: 
1) papers in some fields are typically more cited 
than papers in other fields; 2) old papers natu- 
rally tend to have more citations than more recent 
ones. These facts are part of the common wisdom 
of people working in science, yet it is common to 
see comparisons between the number of cites of 
papers dealing with completely different topics or 



published in different decades. However, as we will 
show, one should not take for granted that a paper 
which has been cited twice as much as another in 
a different field has actually had a larger impact. 
The most natural way to get rid of (or at least 
alleviate) these problems is the use of relative in- 
dicators, i.e. the normalization of the number of 
citations by some average over a suitable reference 
set. The use of relative indicators has been pro- 
posed long ago [14] and used in various contexts, 
but often in questionable ways [15J[T6] and without 
an empirical check that it actually solves the prob- 
lem. Recently we have shown that such a normal- 
ization strongly reduces biases when papers from 
entirely different disciplines, spanning all fields of 
science, are compared [T7J [18]. Here we present 
a similar analysis applied to publications of the 
American Physical Society (APS). With respect to 
Ref. [T71 [18] we consider a more fine-grained level 
of categorization, since the different groups we deal 
with are all within the realm of Physics. Fields are 
identified using the PACS (Physics and Astron- 
omy Classification Scheme [1.9 J number scheme. 
We show first that relevant differences in citation 
patterns show up among different fields. The av- 
erage number of citations in some field can be up 
to three times larger than for papers in another 
field. Analogously, we show that papers published 
in 1985 are cited in some fields even 15 times more 
than papers published in 2009. We then show that 
introducing a suitable relative indicator, the distri- 
butions of such indicator become essentially inde- 
pendent from the field or the year of publication, 
so that the relative indicator can be used as an un- 
biased measure of impact. We corroborate these 
results by showing that ranking papers based on 
the raw number of citations leads to large biases in 



PACS Description 

00 General 

10 The Physics of Elementary Particles and Fields 

20 Nuclear Physics 

30 Atomic and Molecular Physics 

40 Electromagnetism, Optics, Acoustics, Heat Transfer, Classical Mechanics, and Fluid Dynamics 

50 Physics of Gases, Plasmas, and Electric Discharges 

60 Condensed Matter: Structural, Mechanical and Thermal Properties 

70 Condensed Matter: Electronic Structure, Electrical, Magnetic, and Optical Properties 

80 Interdisciplinary Physics and Related Areas of Science and Technology 

90 Geophysics, Astronomy, and Astrophysics 

Table I. The ten categories considered in this study based on the first digit of the first field of the PACS numbers 



favor of some PACS codes, while ranking using the 
relative indicator allows a fair comparison among 
different fields. We make available, for reference, a 
table with the average number of citations for each 
category and each publication year since 1985. We 
finally present further evidence of the relevance of 
the rescaling procedure by considering all authors 
who published in APS journals between 1985 and 
2006. There is of course a correlation between the 
number of raw citations of an author and its/her 
relative indicator, but large deviations are possi- 
ble. The problem is even more remarkable when 
the relative indicator includes a rescaling with the 
number of authors of each paper, so that the num- 
ber of citations of a paper are equally split among 
all co-authors. This highlights that rescaling cites 
with the number of authors is a crucial issue in 
citation analysis. 
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Figure 1. (Color online) Value of the average num- 
ber of cites per paper Co vs publication year for the 
different categories. 



II. DATA SETS 

We consider all papers published in jour- 
nals of the American Physical Society (APS, 
www.aps.org) from 1985 to 2009. We restrict 
our analysis only to standard research publica- 
tions (Letters, Rapid communications, Brief Re- 
ports and Regular Articles) and exclude other type 
of published material (Editorials, Reviews, Com- 
ments, Replies and Errata) which may show dis- 
tinct citation patterns. The journals considered in 
our analysis are: Physical Review Letters, Phys- 
ical Review A, Physical Review B, Physical Re- 
view C, Physical Review D and Physical Review E. 
APS journals represent the most important publi- 
cation outlets in Physics and cover all sub-fields 
of this discipline. They therefore represent an 
optimal benchmark for the study of citation pat- 
terns of publications within Physics [21J. The first 
year of the temporal range considered has been 
selected because in 1985 the PACS coding started 
to be systematically used. We consider only pa- 



pers classified according to the PACS codes, which 
are the vast majority (> 95% between 1985 and 
2009) of all papers published in APS journals. 
PACS numbers are attributed to papers by au- 
thors themselves. This guarantees an optimal 
classification into fields, overcoming the nontriv- 
ial problem of attributing, a posteriori, papers to 
fields [I5JI22]- PACS codes are composed of three 
fields XX.YY.Z Z where the first two are numeri- 
cal (two digits each) and the third is alphanumer- 
ical. For our purpose we consider only the first 
digit of the XX code, which provides a classi- 
fication into very broad categories (see Table . 
Hence, for example, two papers with PACS codes 
05.70.Ln and 02.50.2r both belong to the category 
00, while a paper with PACS number 64.60.ffi is 
part of the category 60. In general, authors as- 
sign to a paper two or three PACS numbers. In 
our analysis we classify papers only according to 
their principal PACS number. For each paper the 
number of cites is obtained from the WebOfScience 
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(WOS, www.isiknowledge.com) database, hence 
including also citations from all other non-APS 
publications included in the WOS database. The 
data collection was performed on Dec. 15, 2010. 
In our analysis we consider only papers which have 
received at least one citation until the above men- 
tioned date. 



III. THE RELEVANCE OF THE 
PROBLEM. MEAN VALUES AND 
DISTRIBUTIONS 




In Fig. [T] we report, for each category, the num- 
ber Co of citations received on average by each pa- 
per as a function of the publication year. It turns 
out that there are quite large differences in the val- 
ues of Co depending both on the category consid- 
ered and on the year. In particular, articles pub- 
lished in the same year in a given field can be cited 
on average up to three times more than in another 
field (e.g. PACS 40 vs PACS 50 in 1985). The av- 
erage number of citations received tends to grow 
as older publications are considered, with rather 
limited fluctuations, with the notable exception of 
the very high peak visible in 1988 for PACS 30, due 
to the extraordinary popularity (more than 30 000 
citations so far) of a single article [23]. In general, 
it is possible to see that the differences between 
the cos of papers published under the same PACS 
number but in different years can be very large: 
for PACS 80 for example, Co — 80 for articles pub- 
lished in 1986 but only Co — 8 for publications of 
2008. 

These data indicate that, although the varia- 
tions are less pronounced with respect to cross- 
discipline comparisons [T71 fT5] . there are impor- 
tant caveats when trying to compare the number 
of citations accrued by articles published in APS 
journals with different PACS codes or publication 
years. 

The same problem is found when one does com- 
pare not only the average values but the full distri- 
butions. For example, in Fig. [2] we plot the cumu- 
lative distribution functions (cdf) of the number 
of citations for articles in the category 70 for var- 
ious publication years. It is evident that over the 
years the distributions change considerably. For 
instance, more than 200 citations is not an uncom- 
mon result for a paper published in 1985, while it is 
an extraordinary achievement for one published in 
2005. Similarly, citation distributions are incom- 
patible also when one considers the same publica- 
tion year but different PACS categories, as shown 
in the inset of Fig. [2] for year 1995. 



Figure 2. (Color online) In the main panel, we plot 
the probability P (> c) to observe a paper with more 
than c citations for category PACS 70 and years, from 
top to bottom: 1985 (black), 1990 (red), 1995 (blue), 
2000 (orange), 2005 (green), 2007 (brown) and 2009 
(turquoise). In the inset, we plot P (> c) for papers 
published in the same year 1995 but in different PACS 
categories: 00 (black), 10 (red), 20 (blue), 30 (orange), 
40 (green), 50 (brown), 60 (turquoise), 70 (magenta), 
80 (gray) and 90 (violet). 



IV. RELATIVE CITATION RATES AND 
UNBIASED RANKING 

The most natural solution to the problem 
pointed out in the previous section is the use of 
relative citation numbers. Let us define the ratio 

c / = f 
Co 

This quantity measures the success of a paper, in 
terms of citations received, compared with other 
papers in the same category and year. A value 
Cf > 1 (< 1) indicates that the paper has been 
cited more (less) than the average. By definition 
the average value of Cf is 1, for any category or 
year, but this is not enough to have a useful un- 
biased indicator. We need the full distributions to 
be the same, so that the probability of having any 
value of Cf is the same, no matter the category 
or the year. In Fig. [3] we show the distributions 
of relative citation numbers for the same data of 
Fig. [2} The comparison between the two figures 
reveals that the normalization procedure rescales 
very well all distributions on top of each other, 
thus allowing a fair comparison among publica- 
tions in different fields and/or years. Some vari- 
ation shows up only for recent publication years 
(> 2005) for which citation patterns are generally 
far from stationary [6 . But also in these cases the 
difference with respect to the asymptotic shape 
of the scaled distribution is small. The scaling 
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evidence that the ranking is biased is found when 
c is used. 

We show in Figs. [6j [7] and [8] the results obtained 
for different publication years, but fixed PACS cat- 
egories. 

The results presented in the figures demonstrate 
that the ranking based on Cf is fair, the fluctu- 
ations around the expected value z% being ac- 
counted for by finite sample effects. On the con- 
trary, when the raw number c of citations is used, 
some sets are largely overrepresented and others 
underrepresented in the top 5% group. Similar re- 
sults (not shown) are obtained for other values of 
z. 



Figure 3. (Color online) In the main panel, we plot 
the probability P (> c/) to observe a paper with rel- 
ative citation count larger than Cf for category PACS 
70 and years 1985 (black), 1990 (red), 1995 (blue), 
2000 (orange), 2005 (green), 2007 (brown) and 2009 
(turquoise). In the inset, we plot P (> Cf) for papers 
published in the same year 1995 but in different PACS 
categories: 00 (black), 10 (red), 20 (blue), 30 (orange), 
40 (green), 50 (brown), 60 (turquoise), 70 (magenta), 
80 (gray) and 90 (violet). 

procedure is able to remove the bias almost com- 
pletely also for papers published one year ago. 
A more quantitative assessment of the improve- 
ment induced by the use of Cf instead of the raw 
number of citations c comes from the ranking of 
articles. We consider papers belonging to several 
sets and rank them according either to the num- 
ber of citations c or to the rescaled indicator cj. 
We compute then the percentage of publications 
of each category that appear in the top z% of the 
global rank. If the ranking is fair the percentage 
for each category should be around z% with small 
fluctuations. 

Fig. [4] clearly shows that when articles are 
ranked according to the unnormalized number of 
citations c there are wide variations among fields. 
Such variations are dramatically reduced when the 
relative indicator Cf is used. More quantitatively, 
assuming that the distributions for the various 
fields are the same, the expected value of the bin 
height in Fig. [4] is z% with a standard deviation 



*(ioo-*) v^ 1 

where N c is the number of categories and Ni the 
number of articles in the i-th category. When the 
ranking is performed according to Cf = c/cq we 
find (Fig. |5| a very good agreement with the hy- 
pothesis that the ranking is unbiased, while strong 



V. RENORMALIZED CITATION COUNTS 
FOR AUTHORS 

In this section we present a practical applica- 
tion to the comparison of individual researchers, 
showing how the rescaled indicator Cf is corre- 
lated but far from being equivalent to the raw 
number of citations c. We have considered all 
papers published in APS journals from 1985 to 
2006 and labeled with PACS numbers, and indi- 
cated, for the generic paper z, with c(i) and Cf(i) 
the number of citations received and the relative 
indicator, respectively. We have then identified, 
for each author a, the set {a} of her/his publica- 
tions]^] and computed the total number of her/his 
citations = X!ie{a} c (0 an d the correspond- 
ing total value of the her/his relative indicator 
Cj, a ) = ^2 ie ^ a y Cf(i). Fig. 9 reports for each au- 
thor the value of the total rescaled indicator Cf 
as a function of the total number of raw citations 
C. The correlation is very good r « 0.98, but 
not perfect. For a single value of one indicator 
there are often values of the other spanning one 
order of magnitude, indicating again that authors 
with a very different number of raw citations may 
be equivalent once differences across subfields and 
time are considered. An even more striking result 
is obtained when the relative indicator is further 
rescaled by the number of authors for each paper: 

{a} C f( i )/ N ( i )^ where N(i) is the num- 
ber of authors of the i-th paper. This rescaling 
is aimed at taking into account multiple author- 
ship [25ti27] and it is based on the assumption 
that all authors contribute equally to each pub- 
lication In this case, the range of variations of 



1 This procedure is subject to many potential errors in the 
identification of authors. However, this problem is not 
big 24]. 

2 This is for sure largely incorrect in many cases, but it is 
the most reasonable assumption as long as precise state- 
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Figure 4. (Color online) Histograms representing the percentage of papers belonging to the top 5% of the global 
rank performed according to c (left column) or c/ (right column). In each panel we consider only papers published 
in a given year and plot the percentage of papers with given PACS category belonging to the top 5% of the global 
rank. Black dashed lines represent the theoretically expected values (in case of a fair ranking) of the mean, while 
the gray areas cover the regions corresponding to the mean ± one standard deviation. 
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Figure 5. (Color online) Comparison between the the- 
oretical values (black line) for the mean (top panel) 
and standard deviation (bottom panel) of bins height 
in Fig. ^] and the values obtained when the ranking is 
performed based on c (red circles) or c/ (blue squares). 



Bf for each value of C is much increased (the r- 
value is reduced to r = 0.49), reaching in many 
case two orders of magnitudes. 



Fig. 



ments about individual contributions are not be pub- 
lished for each article. 



10| leads to 

the striking observation that a researcher with 10 4 
citations may have a number of rescaled citations 
equal to a colleague whose publications have been 
cited only less than 100 times! 



VI. CONCLUSIONS 

In this paper we have investigated the possibility 
to compare in a fair manner the citations of papers 
published in APS journals in different years and/or 
in different fields of Physics. We have shown that 
the raw number of citations is not a suitable in- 
dicator, since there are remarkable differences de- 
pending on the field and on the year of publication. 
A fair comparison is obtained instead if the relative 
number of citations Cf (i.e., the number of cita- 
tions divided by the average number of cites for the 
same category and year) is considered. The nor- 
malization rescales essentially all distributions on 
top of each other and this is further confirmed by 
checking that ranking papers according to Cf does 
not introduce any bias. For completeness, we have 
performed (but not shown here) the same type of 
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Figure 6. (Color online) Histograms representing the percentage of papers belonging to the top 5% of the global 
rank performed according to c (left column) or c/ (right column). In each panel we consider only papers with 
a given PACS number and plot the percentage of papers published in certain year belonging to the top 5% of 
the global rank. Black dashed lines represent the theoretically expected values (in case of a fair ranking) of the 
mean, while the gray areas cover the regions corresponding to the mean ± one standard deviation. 
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Figure 7. Same as fig. [6] but for the remaining set of PACS numbers. 
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Figure 8. (Color online) Comparison between the the- 
oretical values (black line) for the mean (top panel) 
and standard deviation (bottom panel) of bins height 
in Fig. [6] and the values obtained when the ranking is 
performed based on c (red circles) or c/ (blue squares). 
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Figure 9. (Color online) Scatter plot of the values of 
C and Cf for each author who published in APS jour- 
nals between 1985 and 2006. For clarity, data have 
been logarithmically binned and for each bin we plot 
the values of Cf corresponding to the 90% confidence 
intervals (error bars), 25% confidence intervals (boxes) 
and median (horizontal bars inside boxes). 



analysis by using as renormalization factor the me- 
dian value instead of the average. The median is 
less sensitive to possible extreme events such as 
the presence of highly cited papers, but diving the 
raw number of cites by the median value leads to 
less fair comparisons and only for sufficiently old 
publications (< 2000). 

In this paper we have considered fields as iden- 
tified by the first digit of the first number in the 
PACS code. This classifies all papers in Physics in 



Figure 10. (Color online) Scatter plot of the values 
of C and Bf for each author who published in APS 
journals between 1985 and 2006. For clarity, data have 
been logarithmically binned and for each bin we plot 
the values of Bf corresponding to the 90% confidence 
intervals (error bars), 25% confidence intervals (boxes) 
and median (horizontal bars inside boxes). 



10 very broad categories. In principle one can pur- 
sue further this line of investigation, considering 
more refined levels of categorization. A natural 
next step would be the consideration of 100 dis- 
tinct categories, each identified by the whole first 
field of the PACS code. However, the number of 
papers published each year in each of these cat- 
egories is typically very small, and this gives rise 
to huge fluctuations that do not allow to extract 
reliable conclusions. 

We believe that these results are very impor- 
tant in view of the increasing trend towards quan- 
titative evaluation of research performance. We 
strongly encourage researchers dealing with such 
issue to consider relative citation numbers as the 
basis of all their evaluations. All indicators for 
sets of publications (individual authors, groups, 
research institutions) must be constructed based 
on the relative citation numbers. This is impor- 
tant also for very large sets (e.g., at the institution 
level) in order to weigh in a balanced manner the 
contribution of all fields. For this reason the val- 
ues of the average number of citations cq for each 
category and each year will be available at the web 
page filrad.homelinux.org/resources, where 
they will be updated periodically. 

APS journals are an important but clearly par- 
tial domain of the whole range of dissemina- 
tion outlets available for researchers in Physics. 
The extension of the work presented here to in- 
clude also all other journals where research about 
Physics is published is a much needed step to- 
ward a more reliable citation-based research per- 
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formance evaluation. Finally, let us stress that the 
attribution of citations of multi-authored papers 
to individual contributors is a crucial and much 
overlooked issue. Different ways of dealing with 
this problem lead to completely different results, 
as Figs. [9] and 10 demonstrate. The current com- 
mon habit of attributing all citations to all au- 
thors, with no normalization, is unfair and it en- 
courages the misconduct of inflating author lists 
with individuals who did not actually contribute 



to the work. 
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